A Novel Approach: Tokenization Framework based on Sentence Structure in Indonesian Language

This study proposes a new approach in the sentence tokenization process. Sentence tokenization, which is known so far, process of breaking sentences based on spaces as separators. Space-based only generates single word tokens. In consisting five words, will produce tokens, one each. Each token. ignores loss original meaning separated words. Our proposed framework can generate one-word tokens and multi-word at same time. The carried out by extracting structure to obtain elements. element There are elements that Subject, Predicate, Object, Complement Adverbs. We extract structures using deep learning methods, where models built training datasets have been prepared before. results quite good with an F1 score 0.7 it still possible improve. similarity topic for measuring performance compared this case multiword token has better accuracy. was created Indonesian language but also use other languages dataset adjustments.